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Abstract An overload condition can lead to high stress for an operator and further cause substantial drops in performance. On 
other extreme, in automated systems, an operator may become underloaded: in which case, it is difficult for the operator to mainta 
sustained attention. When an unexpected event occurs, either internal or external to the automated system, a disengaged ope rati 
may neglect, misunderstand, or respond slowly /inappropriately to the situation. In this paper, we discuss a systematic approach 
monitor for extremes of cognitive workload and engagement in multitasking environments. Inferences of cognitive workload ar 
engagement are based on subjective evaluations, objective performance measures, physiological signals, and task analysis result 
The systematic approach developed in this paper aggregates these types of information collected under the multitasklr 
environment and can provide a real-time assessment of engagement. 


1. ©INTRODUCTION 

Human operators piay an important role in 
aviation and other safety critical missions, in 
existing aviation systems, the Operator 
Functional State (OFS) is usually not 
monitored and remediation is not 
implemented, in practice, two types of 
hazardous states of awareness are likely to 
lead to human errors [1]: a stress state due 
to high cognitive workload {we do not 
consider physical workload in this research) 
or a compfacent/bored state in extremely 
low workload situations for a prolonged 
period of time [2], it has been found in 
existing research that proper assessment of 
the cognitive workload and appropriate task 
mitigation in overload conditions offers 
potential to improve mission effectiveness 
and aviation safety [3]-[5], 

On the other hand, the disengaged state 
developed in low workload conditions has 
not received equal attention. 
Disengagement is usually accompanied by 
poor situational awareness, which can lead 
to severe consequences in the multi-tasking 


aviation domain. This is especially true in 
typical commercial flight scenario, whic 
has periods of high workload during pre 
flight preparations, takeoff and landing wit 
long periods of very low workload as th 
pilot cruises enroute toward the destinatio 
with the aircraft on autopilot. Pilots ca 
easily get disengaged during the enrout 
phase as they may be less attentive unde 
low workload. When unexpected event 
occur, the disengagement in the tasks bein 
performed could lead to operational errors 
Such events could include unexpecte 
changes in weather {turbulence, fc 
example), equipment faiiure/malfunctio 
(such as hydraulic pump failure) or potentis 
collisions with other aircraft. 

Therefore, the primary focus of thi 
research is to provide a real-tim 
engagement/disengagement assessmer 
mechanism. For this purpose, we will sta 
with a study of the relationship amon 
workload, engagement and performanc 
and identify the causes of low engagemer 
status (low workload) and its effect 
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(impaired performance). To better train an 
engagement assessment model, we will 
design a mechanism to identify the 
engagement ground truth based on different 
sources of information (performance 
measures, subjective evaluation, 
physiological signals, and task/workload 
analysis), followed by a committee machine- 
based real-time assessment model 
technique, and demonstrate the concept 
with a hypothesized dataset. 

The remainder of the paper is organized as 
follows. Section 2 describes the relationship 
among workload, engagement, and 
performance. Section 3 presents a 
mechanism to determine the ground truth 
for engagement modeling. Section 4 
describes an enhanced committee machine- 
based real-time engagement assessment 
model. Section 5 shows preliminary 
simulation results. Section 6 concludes the 
paper. 

2.0WORKLOAD, ENGAGEMENT, AND 
PERFORMANCE 

The relationship between task engagement 
and performance has been an active area of 
research for over 20 years [6], Researchers 
describe a state of high performance called 
“flow,” which occurs when people are 
performing challenging tasks for which they 
have high skill. When a person’s skill 
exceeds the task challenge, it is very likely 
he/she may become bored and disengaged. 
Performance can suffer if the imposed 
workload is greater than the resource that 
an operator can afford. 


performance will accordingly decrease after 
a certain amount of time, which will be the 
focus of this paper. 

Therefore, performance can usually be 
affected by both workload and engagement. 
We define performance as a function of 
imposed workload (WL /mposed ), workload 
capacity (WL C ), engagement (£), and 
efficiency ( E n ). These terms are defined as 
follows: 

• Imposed workload is typically what is 
provided to the operator and consists of 
what objectives need to be met, a period 
of performance (e g., a deadline or a 
length of time an activity must continue), 
criteria for success or quality (i.e., how 
the work may be evaluated), and other 
constraints that apply (e.g., what 
resources or people a person has 
available or whether a failure occurs in 
the system and how an operator is 
qualified). 

• Workload capacity of an individual can 
change due to physical fitness (sleep 
loss, sickness, etc.) or training. 

• Engagement is how much attention an 
operator puts in a task. 

• Efficiency is usually determined by how 
efficiently he/she accomplishes a task. 

The ratio between the imposed workload 
and the workload capacity basically 
determines whether an operator is either 
overloaded or underloaded: 


Workload and engagement are closely 
related. An optimal workload, one in which 
an operator performs challenging tasks 
within his or her abilities, leads to high 
levels of engagement and, accordingly, high 
levels of performance. If workload exceeds 
an operator’s capacity, he/she will be 
overloaded and the performance will drop 
eventually. By the same token, if an 
operator is under a low workload condition 
for a prolonged period of time, he/she will 
usually drift into a disengaged state and the 


a = 


^^Imposed 

WL C 


(Equation 1) 


If a is beyond 1 , the task requirements are 
greater than the person’s processing 
capacity, and his or her workload is high; 
whereas if a is at or around 1 , workload is 
appropriate for the worker. However, if a is 
well below 1 (for example < 0.5), workload 
is too low for the operator. 

With the terms defined, performance can be 
derived based on the difference between 
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the imposed workload and the effective & 
engaged processing workload (WL EE ), 
which is determined by the capacity, 
engagement and efficiency: 

WL ee = WL c *E*Eti (Equation 2) 

If the imposed workload WLi mp0S ed is greater 
than the effective and engaged processing 
workload WLee, the performance can be low 
due to the insufficient instantaneous 
processing power to meet the task 
requirements; otherwise, performance can 
be satisfactory. 

Although performance can be affected by 
these four factors (capacity, engagement, 
efficiency, and imposed workload); in 
practice, real-time performance variation is 
not likely due to efficiency changes since 
experienced operators generally use 
efficient strategies. Also, there is a plethora 
of research on workload capacity (such as 
fatigue) and overload conditions. Therefore, 
in this research, we only focus on the study 
of engagement under low workload and task 
performance is used as an indicator of the 
engagement state. 

3.0 ENGAGEMENT GROUND TRUTH 

Before an engagement assessment model 
can be deployed, it needs to be trained 
based on the engagement ground truth and 
corresponding input information 
(physiological signals, performance, and 
others). However, there does not exist a 
sensor to provide engagement ground truth; 
instead, engagement ground truth is often 
derived based on all available information, 
including workload analysis, subjective 
evaluations, performance measures, and 
physiological measures. In each of the four 
types of information, different characteristics 
exist. 

First, as discussed before, disengagement 
usually occurs under low workload 
conditions, and therefore, workload 
information shall be utilized to assess the 
engagement state. Cognitive workload 
analysis under multi-tasking environments 
can provide a direct and continuous 


measure of the tasks being performed. We 
can hypothesize that an operator in a low 
workload condition for a prolonged period of 
time may become disengaged. It is 
important to note that the cognitive workload 
analysis is an objective measure of the task 
requirements and it cannot account for 
many other factors, such as individual 
variations and environmental conditions. 

Second, subjective evaluation during 
missions is not suitable for identifying 
engagement ground truth (real-time 
assessment also), since in-mission 
subjective evaluation requires interaction 
with the operator being monitored, which 
would affect the operator’s state artificially. 
Instead, a subjective evaluation after the 
mission can be directly utilized to assess 
the engagement state when he/she was 
performing the task. For example, recall of 
task/scenario events and a question of “did 
you feel you were engaged while doing the 
task?" can provide the information whether 
the operator was engaged during the task. 

Third, performance measures can reflect 
the effects of individual characteristics and 
contextual information (including system 
setup, hardware/software issues, etc.) on 
engagement. Similar to in-mission 
subjective evaluation, intrusive performance 
measures, such as the fatigue- related 
Psychomotor Vigilance Test (PVT), a 
sustained attention task requiring subject 
response to an isolated target, is also not 
suitable for engagement assessment. 
However, non-intrusive performance 
measures, such as reaction time to Air 
Traffic Controller (ATC) communications, 
can be used as a good engagement 
indicator; slow response times, requests for 
clarification, and errors in readback could be 
associated with a disengaged state. 

Finally, selected physiological measures 
can indicate when an operator is in a 
disengaged state. For example, a 
disengaged pilot during the enroute phase 
may have longer fixation durations and/or 
increased saccade length due to decreased 


636 



workload [7], Other physiological measures 
include EEG readings, facial analysis, body 
posture, pressure readings from a pressure 
sensitive mouse or other equipment (to 
measure stress levels) and use of a 
wristband to measure stress as well 
Previous research using facial analysis, 
posture, the mouse and the wristband 
shows that these physiological measures 
can correlate up to .78 with subject reports 
of engagement during a task (every five 
minutes [8], Adding EEG could potentially 
strengthen this measure and by adding eye 
tracking, we can get extra evidence as to 
whether the pilot is actually attending to the 
unexpected event. 

Based on the characteristics of these kinds 
of information, the ground truth finding 
procedure can be described as follows: 

1) Analyze cognitive workload for the 
task(s) being performed: 

Outcome: WL imposec i', a continuous 

measure of the cognitive workload (0- 
100) induced by the tasks being 
performed. 

2) Performance evaluation: we will derive a 
performance-based engagement score 
based on collected performance 
measures (continuous and/or discrete; 
for example, a relatively long reaction 
time to the ATC communications would 
probably indicate a disengaged state). If 
only discrete performance measures are 
available, interpolation can be used to 
derive the performance-based 
engagement scores in between 

Outcome: a performance-based 

engagement score (0-100). 

3) Fusion of cognitive task analysis results 
and performance-based engagement 
scores. Different fusion techniques can 
be adopted to combine the cognitive 
task analysis and performance 
evaluation results. A simple example is 
a set of fuzzy rules to fuse the imposed 


cognitive workload and the performance 
loss, such as 

a. If WLjmposed is high, performance 
is high, the engagement score is 
high; 

b. If WLimposed is high, performance 
is low, the engagement score is 
medium; and 

c. If WLj mpose< j is low for certain 
duration and performance-based 
engagement score is low, 
engagement is low. 

Outcome: E 0 ; a continuous objective 
measure of engagement (0-100) 

4) Utilize critical physiological signals and 
features to indicate a disengaged state. 
Please note that this step only identifies 
potential disengaged state indicators 
during a mission, such as yawning and 
long fixation duration. 

Outcome: a discrete objective 

engagement score (E DO , 0-1 00) 
representing how well an operator is 
engaged in a task based on critical 
disengaged physiological signs. 

5) Analyze subjective evaluation results. 
There may be more than one subjective 
evaluation measures. In this case, we 
will first fuse these different discrete 
subjective evaluation results. 

Outcome: E os ; a discrete subjective 
measure of the engagement (0-100) 

6) Calibrate the continuous objective 
engagement ( E 0 ) with the subjective 
engagement assessment ( E s = {E do . 
E D $}) If at the same time instant, both 
Edo and E DS are available, they will first 
be combined before calibration 
(weighted sum, for example). 

Outcome: Engagement (E); a 

continuous overall measure (0-100) 
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E= E a + weight * (E s - E B )*e~ F ™ 

(Equation 3} 


With the above procedures, a continuous 
engagement measure can be derived 
considering different sources of information. 


where t 0 is the time instant, when the 
subjective engagement result is available; 
weight (0-1) is the confidence level of the 
subjective assessment result relative to the 
objective assessment result; and F is a 
forgetting factor that controls how long the 
subjective evaluation assessment result will 
impact the final outcome. 

The goal is to use the subjective 
assessment to enhance the overall 
engagement assessment accuracy. In real 
operations, objective assessment results 
can be obtained frequently depending on 
the computer processing speed. On the 
other hand, subjective assessment results 
can only be available at longer time 
intervals because the assessment process 
is intrusive. In other words, the overall 
engagement mechanism is multi-variate. 
The basic idea behind Equation 3 is as 
follows. In the absence of subjective 
assessment results, we will solely rely on 
the objective results, which are more 
frequently available. When the subject 
assessment is available at t 0 , we will use 
Equation 3 to calibrate the final engagement 
assessment result. Depending on our 
confidence level of subjective assessment, 
we will choose a proper weight or the weight 
can be trained with the training data. If 
weight is 0, E = E 0 , which means that the 
subjective assessment of engagement is 
totally discounted and the engagement 
result fully relies on the objective 
measurements; On the other hand, if the 
weight at the other extreme of 1, E = Es, 
meaning the engagement is determined 
solely by subjective assessment at the time 
the subjective assessment is introduced. 
Even with a confidence level defined 
(weight), to reduce the bias from the 
subjective assessment, we introduce an 
exponential term in Equation 3, with which 
the bias is exponentially discounted 
(controlled by the forgetting factor F), 


4.0REAL-TIME ENGAGEMENT 
ASSESSMENT MODEL 

The basic procedure for real-time OFS 
assessment is shown in Figure 1. 


Physiological 
Features [ 


WL 

Discrete Nor-: 

Intrusrve 

performance 




Objectwe/Contm uolts 
Engagement 
Assessment Model 


=>. 


Objective 

Engagement 

Assessment 

Calibrate 



Discrete- time 
Engagement 
Assessment 


Engagement = E a + weight * (E x -E B )*e f " ff 


Figure 1: Real-time OFS assessment 
procedure 

It is similar to the engagement ground truth 
finding procedure described in Section 3. 
However, in real-time aviation applications, 
we cannot rely on manual selection of 
physiological features that can indicate a 
disengaged state, such as eye fixation 
duration and Heart Rate Variability (HRV). 
Instead, the physiological signals are being 
continuously monitored and the variation of 
engagement is automatically determined by 
a model trained with the physiological 
features and the identified engagement 
ground truth using a set of training data. 
The output of the real-time assessment 
model is an objective assessment of 
engagement. 

Two sources of discrete information are 
utilized to “calibrate 1 ’ the objective 
engagement assessment: the imposed 

workload (especially low workload for a 
prolonged period of time) and non-intrusive 
performance measures (mostly discrete in 
commercial aviation applications, such as 
reaction time and errors associated with 
ATC communications). Fuzzy rules similar 
to those used in ground truth finding can be 
applied to derive a discrete-time evaluation 
of engaged/disengaged state. 
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Again, the final engagement assessment is 
based on a calibration of the objective 
evaluation using the difference between 
objective and subjective evaluation results 
modulated by a forgetting factor. 

It is worth noting that an enhanced 
committee machine method has been 
proposed by the authors in [9], In this 
research, we will apply the same technique 
to build the objective/continuous 
engagement assessment model. The 
enhanced committee machine method is 
able to address large OFS individual 
variations by selecting the committee 
members and features that are the most 
sensitive to the OFS of each individual. The 
method has been successfully verified and 
validated with a driving test data set with a 
mean squared error of OFS estimation 
being significantly decreased (by around 
20%) comparing to that without 
individualization [9], 

5.0 SIMULATION RESULTS 

In this paper, we generated a simulation 
dataset based on the flight information of 
AAL1238 on 05/12/2010, from Seattle to 
Chicago O'Hare, to illustrate the developed 
engagement assessment method. The 
basic flight information was extracted from 
the link from [10], The altitude change along 
the flight is shown in Figure 2, 


x id FH^t franSeattle to a*X5goOH*e(A4L123§ OSMOND 



Figure 2: Flight AAL 1238 on 05/10/2010: 
altitude vs. time 

Several assumptions were made to adopt 
this flight for the proof-of-concept purpose: 


1) The imposed workload was 
constantly high during take-off and 
landing 

2) The imposed workload was low 
during cruising (altitude above 30k 
ft). 

3) Additional assumptions: 

a. Both an engaged pilot and a 
disengaged pilot reported 
disengaged from 1:45PM to 
2:15PM 

b. Reaction time of an engaged 
pilot is shorter; four reaction 
times are assumed: 1, 2, 2.4, 8, 
and 2 seconds vs. 3.2, 6.5, 8.3, 
5.5 seconds for a disengaged 
pilot. 

c. Simulated fixation duration is 
used as a performance indicator 
for engagement assessment 

Figure 3 shows the imposed workload, low 
workload more than 2 minutes, and 
subjective evaluation of disengagement. 



Figure 3: Workload, low workload more than 
2 minutes, and subjective evaluation of 
disengagement 

For an engaged pilot, the fixation duration is 
usually smaller than that of a disengaged 
pilot, who may be in a state of day dreaming 
or high fatigue. Also, the reaction time of the 
engaged pilot is usually shorter than that of 
a disengaged pilot. As an example, Figure 4 
and Figure 5 show the physiological signals 
(normalized fixation duration) and the 
physiological indicators of a disengaged 
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pilot and an engaged pilot, respectively. It 
can be seen that a few more physiological 
indicators of disengagement are found for a 
disengaged pilot (shown in black). 


Ds*?nr£ig?ii PiW 



Figure 4: Physiological signals and 
physiological indicators of a disengaged 
pilot 


EngaiedPikrf 



Figure 5: Physiological signals and 
physiological indicators of an engaged pilot 


By combining with the performance 
indicators derived from reaction time, we 
derived the engagement score for an 
engaged pilot vs. a disengaged pilot with 
the method described in this paper. An 
example plot of the final engagement scores 
is shown in Figure 6. 



<y — i * 1 1 

1200 iaoo 14:00 iaoo moo 


Figure 6: Disengagement score of an engaged 
pilot vs. a disengaged pilot 
Clearly, from the example shown above, we 
can see that engagement/disengagement 
state assessment cannot solely rely on the 
subjective evaluation results, and low 
workload does not necessarily indicate a 
disengaged state. Physiological indicators 
of disengagement and selected non- 
intrusive performance measures, although 
discrete, can usually provide a better 
estimation of disengagement. 

6.0CONCLUSION 

In this research, we have successfully 
developed a systematic approach for 
engagement assessment. The approach is 
based on a thorough understanding of the 
relationship among performance, workload, 
and engagement. To train a real-time 
engagement assessment mode!, we have 
developed a systematic approach to identify 
the engagement ground truth based on 
different sources of information: workload, 
non-intrusive performance measures, 
physiological indicators, and subjective 
evaluations. The ground truth identification 
approach was demonstrated using a 
simulation data derived from the AAL1238 
flight on 05/12/2010. 

One of the future tasks is to further 
implement the proposed real-time 
assessment technique on the enhanced 
committee machine-based model and is to 
verify and validate its performance with 
experimental data. Another important task is 
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to continue addressing the individual 
variation in the enhanced committee 
machine-based real-time engagement 
assessment model. 

7.0 ACKNOWLEDGMENTS 

This project was funded by the NASA 
(Contract No: NNX10CB27C). We thank Dr. 
Alan T. Pope, our COTR, for his comments 
and suggestions as we performed this 
research. We also thank Dr. Kara Latorella 
for her comments and feedback during 
various discussions over the course of this 
project. 

8.0REFERENCES 

[lj.Pope, A. I.; and Bogart, E. H. (1993). 
identification of Hazardous Awareness 
States in Monitoring Environments, SAE 
Technical Paper No. 921136, SAE 1992 
T ransactions: Journal of Aerospace, 
Section 1 - Volume 101, pp. 449-457. 

[2] , Paul C. Schutte, et.al. “The Naturalistic 

Flight Deck System: An Integrated 

System Concept for Improved Single- 
Pilot Operations,” NASA report, 
December 2007 

[3] .Schmorrow D, Kruse AA. Improving 
human performance through advanced 
cognitive system technology. 
Interservice/ Industry Training, 
Simulation and Education Conference, 
Orlando, FL; 2002. 

[4] . Sch morrow D, Stanney KM, Wilson GF. 
Young P. Augmented cognition in 
human-system interaction. In: Salvendy 
G, ed. Handbook of human factors and 
ergonomics, 3rd ed. New York: John 
Wiley; 2005. 

[5] , Karl F. Van Orden; Erik Viirre; David A. 
Kobus, “Augmenting Task-Centered 
Design with Operator State Assessment 
Technologies,” Foundations of 
Augmented Cognition: Lecture Notes in 
Computer Science, 2007, Volume 
4565/2007, 212-219. 

[6] .Csikszentmihalyi, M. and 

Csikszentmihalyi, I.S. (1988). Optimal 
experience: Psychological studies of 


flow in Consciousness. New York: 
Cambridge University Press. 

[7] , Ellis, Kyle Kent Edward, "Eye tracking 

metrics for workload estimation in flight 
deck operations" (2009). Theses and 
Dissertations. Paper 288. 

[8] , Woolf, B„ Burleson, W., Arroyo, I., 

Dragon, T_, Cooper, D and Picard, R. 
(2009) ‘Affect-aware tutors: recognising 
and responding to student affect’, Int. J. 
Learning Technology, Voi. 4, Nos. 3/4, 
pp. 129-164. 

[9] .Guangfan Zhang, Roger Xu, Wei Wang, 
Jiang Li, Tom Schnell, Mike Keller, 
Individualized Cognitive Modeling for 
Closed-Loop Task Mitigation, ModSim 
World Conference, 2009, Virginia 
Beach, VA 

[10] . http://flightaware. com/li ve/flight/AA L 

1 238/history/20 1 005 10/161 OZ/KSEA/KO 

RD/tracklog 


641 



October 13-15, 2010 
Hampton Roads Convention 
Center, Hampton, VA 



MODSIM WORLD 

Conference & Expo 


A Systematic Approach for 
Engagement Analysis 
under Multitasking Environments 


Guangfan Zhang. John Leddo and Roger Xu 
(Intelligent Automation, Inc.) 

Carl Richey and Tom Schnell (University of Iowa ) 
Frederick McKenzie and Jiang Li (Old Dominion University) 


Outline 


• Background 

• Engagement Analysis 

• Enhanced Committee Machine-based 
Engagement Assessment 

• Simulation Results 

• Conclusion & Acknowledgement 


642 


Background 

• Definition of Engagement 

Attentional state of an operator during execution of a given task 

• Two types of hazardous states of awareness that may lead to 
human errors: 

- A stress state due to high cognitive workload 

- A disengaged state due to low workload, poor physical fitness, etc. 


• An operator in a low engagement level may neglect, 
misunderstand, or respond slowly/inappropriately to 
unexpected events 

- For example, commercial flight pilots can easily get disengaged during 
enroute phase under low workload 


Definitions 

• Imposed workload, WL lmposed 

- The workload that is assigned to successfully accomplish the 
given task. 


• Workload capacity, WL C 

- The maximum workload an operator can handle in the task 


• Effective workload, WL EE 

- The workload that the operator actually delivered toward the 
given task 


643 


Engagement and Performance 


• WL cc =WL c *E*E ff 

where E denotes the engagement (a score from 0 to 100), W EE 
the effective workload, WL C the workload capacity, and E ff 
stands for efficiency , which is determined by how efficiently 
the operator accomplishes a task 

• Performance: 

per f = \ 1 If WL w * succeed 

\vL F£ iWL tmpmal otherwise , failed 


Note that 




WLcE^ 

WJ 

^imposed 


Engagement Analysis 

• Engagement is determined by multiple factors 

WL 

- Task challenging level, a = —^± 

- Physical fitness, e g., sleep loss, 'sickness 

- Environmental conditions 

• We focus on the relationship between the 
engagement and imposed workload 


£(«)=/» 

CX denotes the challenging lever, 
and B denotes other factors that 
affect engagement 



644 


Engagement 


Engagement Assessment 


• Awareness evaluation 

- e.g., reaction time to the Air Traffic Controller (ATC) 
communications 

- Continuous and/or discrete 

• Physiological signals and features 

- e.g., EEG readings, eye fixation durations, heart rate variability 
(hfRV), etc. 

- Continuous 

• Subjective evaluation 

- e.g., fatigue-related Psychomotor Vigilance Test (PVT) 

- Discrete 


Overall Engagement Assessment 

• Fusion of continuous objective engagement measure 
E 0 (t) with discrete subjective measure E s (tJ 

E(t) = w 0 E 0 (/) + w s g(E x , t) 



• g(E s ,t) is a prediction 
function based on the 
previous discrete measures 

m 

• £(£>*0 can be estimated 
using a data-driven or 
parametric model method 
in the prediction theory 
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Real-time Engagement Assessment 

Model 


Physiological 
Features 


Wt- im p 0se tf & 

Discrete Non- 

intrusive 

performance 




Objective/Continuous 
Engagement 
Assessment Model 


> 


Fuzzy Logic 


Objective 


Engagement 

Assessment 

t 


Fusion 


0 


Discrete-time 

Engagement 

Assessment 


Engagement = w u E (> (t) + w s g(E s J) 


Enhanced Committee Machine for Objective/ 
Continuous Engagement Assessment 

Enhanced Committee Machine [1] 

* Feature selection + Bootstrapping 

* Advanced Feature Selection: Piecewise Linear 
Orthogonal Floating Search (PLOFS) [2] 

- Computationally Efficient 

• Performance: Wrapper type 

• Speed: Filter type 

- Select from Original Features 

• No transformation needed like PCA 

- Consider interactions among features 

- Generate a list of combinations 

* Bootstrapping: resubmission 
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Enhanced Committee Machine Architecture 



Committee 1: 
Trained with 
different initial 
weights 

Committee 2: 
Bagging 
combined with 
feature 
selection 

Committee 2 
is better 


Simulations Results 

AAL1238 on 05/12/2010, from Seattle to Chicago O'Hare (3] 
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Flight AAL 1238 on 5/12/2010 
altitude vs. time 
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Simulation Results (cont') 


•Simulated datasets 



Workload, low workload more than 2 minutes, 
and subjective evaluation of disengagement 


Simulation Results (cont') 



Normalized eye fixation duration 
and physiological indicators 
of a disengaged pilot 


Normalized eye fixation duration 
and physiological indicators 
of an engaged pilot 
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Simulation Results (cont') 



Engagement score of 
an engaged pilot vs. 
a disengaged pilot 


•Engagement assessment cannot solely rely on the subjective 
evaluation 

•Low workload does not necessarily indicate a disengaged state 
♦Physiological indicators and selected non-intrusive awareness 
measures usually provide a better estimation 


Summary 

• Established the relationship among performance, 
workload and engagement 

• Developed a systematic approach for engagement 
assessment 

• Demonstrated the feasibility of the proposed 
approach with simulations 

• Future work: 

- Verify and validate the proposed system with experimental data 

— Address the individual variation in the enhanced committee 
machine-based real-time engagement assessment model 
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